A Stable High-Order Tuner for General Convex Functions

Iterative gradient-based algorithms have been increasingly applied for the training of a broad variety machine learning models including large neural-nets. In particular, momentum-based methods, with accelerated guarantees, received lot attention due to their provable guarantees fast in certain classes problems and multiple derived. However, properties these methods hold only constant regressors. When time-varying regressors occur, which is commonplace dynamic systems, many cannot guarantee stability. Recently, new High-order Tuner (HT) was developed linear regression shown 1) stability asymptotic convergence 2) non-asymptotic this letter, we extend discuss results same HT general convex loss functions. Through exploitation convexity smoothness definitions, establish similar guarantees. Finally, provide numerical simulations supporting satisfactory behavior algorithm as well an property.